Agenda

• Introduction to Deep Learning
• Introduction to Tensorflow and Keras
• Hands-on with Keras
• Advanced Deep Learning Topics

A Comprehensive Survey on Deep Learning Approaches

See arXiv:1803.01164

1943: McCulloch and Pitts

McCulloch & Pitts show that neurons can be combined to construct a Turing machine (using ANDs, ORs, & NOTs).

1943: McCulloch and Pitts – Turing Machines

1958: Rosenblatt – The Perceptron

The perceptron: A probabilistic model for information storage and organization in the brain

1958: Rosenblatt – Exercise (OR)

\[ f(x) = \begin{cases} 1 & \sum_{i=1}^m w_i x_i + b > 0\\ 0 & \text{otherwise} \end{cases} \]

tt_or <- tibble(
  a = c(0, 0, 1, 1),
  b = c(0, 1, 0, 1),
  r = c(0, 1, 1, 1)
) %>% print()

x <- as.matrix(select(tt_or, a, b))

b <- __
w <- c(__, __)

ifelse(w %*% x + b > 0, 1, 0)

1958: Rosenblatt – Exercise (AND)

\[ f(x) = \begin{cases} 1 & \sum_{i=1}^m w_i x_i + b > 0\\ 0 & \text{otherwise} \end{cases} \]

library(tidyverse)
tt_and <- tibble(
  a = c(0, 0, 1, 1),
  b = c(0, 1, 0, 1),
  r = c(0, 0, 0, 1)
) %>% print()

x <- as.matrix(select(tt_and, a, b))

b <- __
w <- c(__, __)

ifelse(w %*% x + b > 0, 1, 0)

1958: Rosenblatt – Demo

Rosenblatt, with the image sensor of the Mark I Perceptron (…) it learned to differentiate between right and left after fifty attempts.

1958: Rosenblatt – Predictions

Expected to walk, talk, see, write, reproduce itself and be conscious of its existence, although (…) it learned to differentiate between right and left after fifty attempts.

1958: Rosenblatt – Principles of Neurodynamics 1/3

1958: Rosenblatt – Principles of Neurodynamics 2/3

1958: Rosenblatt – Principles of Neurodynamics 3/3

1969: Minsky and Papert – Book (1/3)

1969: Minsky and Papert – Book (2/3)

1969: Minsky and Papert – Exercise (XOR)

\[ f(x) = \begin{cases} 1 & \sum_{i=1}^m w_i x_i + b > 0\\ 0 & \text{otherwise} \end{cases} \]

tt_xor <- tibble(
  a = c(0, 0, 1, 1),
  b = c(0, 1, 0, 1),
  r = c(0, 1, 1, 0)
) %>% print()

x <- as.matrix(select(tt_xor, a, b))

b <- __
w <- c(__, __)

ifelse(w %*% x + b > 0, 1, 0)

1969: Minsky and Papert – Book (3/3)

It ought to be possible to devise a training algorithm to optimize the weights in this using (…) we have not investigated this.

1969: Minsky and Papert – Solution (XOR)

tt_xor <- tibble(
  a = c(0, 0, 1, 1),
  b = c(0, 1, 0, 1),
  r = c(0, 1, 1, 0)
) %>% print()

w <- matrix(c( 0.5,  0.5,    0,
              -0.5, -0.5,    1,
               0.5,  0.5, -0.5), ncol = 3, byrow = T)
              
yh1 <- ifelse(x %*% w[1,] > 0, 1, 0)
yh2 <- ifelse(x %*% w[2,] > 0, 1, 0)

x3 <- matrix(c(yh1, yh2, c(1,1,1,1)), ncol = 3)
ifelse(x3 %*% w[3,] > 0, 1, 0)